Increasingly, still and motion images are recorded in digital form. Digital still and motion images can be captured using a digital still or digital video cameras. Digital still and motion images are also obtained by converting images that have been recorded in other ways into digital form. For example it is well known to use analog to digital converters to convert analog electronic video signals into digital images. It is also known to use optical scanners to derive digital images from images recorded on photographic prints, films, plates and negatives.
Digital still and motion images are easily viewed, stored, retrieved, and printed by a user using a home computer or other image processing device. Such images can be uploaded to a website for viewing, as described in commonly assigned U.S. Pat. No. 5,666,215 filed by Fredlund et al. on Aug. 3, 1995. Using a web browser, uploaded digital images can be viewed, selected for printing, electronically transmitted to other family members and/or friends or stored in on-line databases and photo albums.
With the recent increase in the use of digital cameras for picture taking and with the recent growth in the use of technology that converts conventional still images, analog video images, and film based motion pictures into digital form, the volume of digital images that are available is rapidly increasing. However, users often do not immediately print or otherwise use the digital images, but instead opt to upload digital images to an electronic storage device or data storage medium for later use. Accordingly, personal computers, personal digital assistants, computer networks, optical, magnetic and electronic storage mediums, so-called set top television devices and other electronic image storage devices are increasingly being used to store digital still and motion images.
Therefore the task of classifying or cataloging digital still and motion images on such storage devices in a way that they will be easily accessible by the user is becoming increasingly important. Some users create large personal databases to organize the digital still and motion images on such storage devices. Many computer programs have been developed to help users to do this. However, because of the time and effort necessary to review and categorize images, these databases are typically only rarely used and updated.
Thus, what is needed is a way to help organize and categorize images with reduced emphasis on the post capture analysis and categorization of images.
Even when users make the investment of time and energy necessary to organize images into databases, the databases are typically organized according to various categories such as the date of capture, places, events, people. Other categories are also used. Often, such categories do not inherently help the user to locate images that are of particular importance or value. Instead the user must remember the image, and when the image was captured and/or how the user categorized it.
Thus, what is also needed is a more useful basis for organizing images. It is known from various studies and observations that the most memorable categories of events and subsequently pictures are the ones that are associated with user's feelings at the time of capture or the emotional reaction that the user experienced during the event or at the scene. Information that can be used to characterize the emotional state of a user is known as affective information. Affective information represents a user's psychological, physiological, and behavioral reactions to an event. Affective information can refer both to recorded raw physiological signals and their interpretations. Using affective information, digital still and video images can be classified based on a user's subjective importance, a degree of preference or the intensity of and nature of specific emotions. Such classifications can help to quickly find, review and share those valuable images.
Various methods are known in the art for deriving affective information based upon a user's reaction to an image. One example of a system that monitors physiological conditions to derive affective information is a wearable capture system that enables the classification of images as important or unimportant based on biosignals from human body. This system was described in an article entitled “Humanistic Intelligence: “WearComp” as a new framework and application for intelligent signal processing” published in the Proceedings of the Institute of Electrical and Electronics Engineers (IEEE), 86, pp. 2123–2151, 1998 by Mann. In his paper, Mann described an example of how the system could potentially operate in a situation when a wearer was attacked by a robber wielding a shotgun, and demanding cash. In this case, the system detects physiological signals such as a sudden increase of the wearer's heart rate with no corresponding increase in footstep rate. Then, the system makes an inference from the biosignals about high importance of the visual information. This, in turn, triggers recording of images from the wearer's camera and sending these images to friends or relatives who would determine a degree of a danger.
Another example of such a system is described in a paper entitled, “StartleCam: A Cybernetic Wearable Camera” published in: Proceedings of the Second International Symposium on Wearable Computers, 1998, by Healey et al. In the system proposed in this paper, a wearable video camera with a computer and a physiological sensor that monitors skin conductivity are used. The system is based on detecting a startle response—a fast change in the skin conductance. Such a change in the skin conductance is often associated with reactions of sudden arousal, fear or stress. When the startle response is detected, a buffer of digital images, recently captured by the wearer's digital camera, is saved and can be optionally transmitted wirelessly to the remote computer. This selective storage of digital images creates a “memory” archive for the wearer which aims to mimic the wearer's own selective memory response. In another mode, the camera can be set to automatically record images at a specified frequency, when very few responses have been detected from the wearer, indicating that their attention level has dropped.
The systems proposed by Mann et al. make use of the physiological signals to classify images as “important” (i.e., causing rapid change in a biological response) or “unimportant” (i.e., not causing rapid change in a biological response), and trigger the wearable camera to store and/or transmit only the “important” images. However, their systems have several shortcomings.
The described systems do not associate, do not store, and do not transmit the physiological signals, or any other “importance” identifier together with the corresponding images. As a result, the “important” images can be easily lost among other images in a database, since there is nothing in these “important” images to indicate that these images are “important”. This can happen, for example, when the digital image files are used on a different system, when the images are transferred via a recordable contact disk or other media, when the images are uploaded to an on-line photo service provider, etc. The described systems also do not associate, do not store, and do not transmit the user's identifier together with the corresponding images. Therefore, when the system is used by more that one user, it is unable to distinguish which user reacts to the image as “important”.
Further, the described systems provide only binary classification “important-unimportant” and do not allow a finer differentiation of the relative degree of importance between the captured images. As a result, after a certain time of acquiring images in the user's database, the number of important images becomes too large to serve the purpose of the importance attribute, unless the user will change the attribute for every image in his or her database, which is a lengthy and tiresome process.
Additionally, the described systems provide image classification only based on the general “importance” attribute. For example, they are unable to differentiate whether the important image evoked a positive (happy) or negative (unhappy) reaction in the user. Therefore, a wide range of human emotional reactions (e.g., joy, sadness, anger, fear, interest, etc.) is not considered in the system and cannot be applied to the advantage of the user.
Consequently, a further need exists for an improved method for obtaining affective information and for using the affective information to facilitate storage and retrieval of images.